Detection system for assisting a driver when driving a vehicle using a plurality of image capturing devices

ABSTRACT

A detection system ( 5 ) for assisting a driver when driving a vehicle, the system ( 5 ) comprising: a plurality of image capturing devices ( 98 ) mounted to the vehicle to capture images of the external environment of the vehicle; an image processing module ( 95 ) to identify potential hazards in real-time from the captured images and to superimpose information of the external environment relating to the identified potential hazards; and at least one display device ( 92 ) to display images processed by the image processing module in real-time; wherein the superimposed information is provided in an area of the display device ( 92 ) to ensure visual clarity of the images of the external environment.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a division of U.S. patent application Ser. No.12/495,508 filed Jun. 30, 2009, and titled “Detection System forAssisting a Driver When Driving A Vehicle Using A Plurality Of ImageCapturing Devices”, the entire disclosure of which is herebyincorporated by references as if fully set forth herein.

TECHNICAL FIELD

The invention concerns a detection system for assisting a driver whendriving a vehicle.

BACKGROUND OF THE INVENTION

Side view mirrors increase the drag coefficient of a vehicle.Consequently, fuel consumption is increased.

Many factors may improve driving safety including an increased Sever ofawareness of potential hazards while driving, in certain environmentalconditions, it is desirable to enhance or augment visual informationprovided to the driver to highlight the existence of potential hazardsand direct the focus of the driver in order to maintain safe driving.

SUMMARY OF THE INVENTION

In a first preferred aspect, there is provided a detection system forassisting a driver when driving a vehicle. The system includes aplurality of image capturing devices mounted to the vehicle to captureimages of the external environment of the vehicle. The system alsoincludes an image processing module to identify potential hazards inreal-time from the captured images and to superimpose information of theexternal environment relating to the identified potential hazards. Thesystem also includes, at least one display device to display imagesprocessed by the image processing module in real-time. The superimposedinformation is provided in an area of the display device to ensurevisual clarity of the images of the external environment.

The image processing module may include video extraction logics toextract and identify potential hazards, the video extraction logicsdetecting the variations between successive frames captured by theplurality of image capturing devices.

The system may further include a camera for tracking eye movement andhead movement in order to detect the behaviour of the driver inanticipation of intention, and the detected behaviour is used by theimage processing module to determine objects of concern that arehighlighted to the driver on the at least one display device.

The system may further include a switch operatively connected to asteering wheel of the vehicle to detect the abnormal drift of thevehicle, and the detected abnormal drift is used by the image processingmodule to determine objects of concern that are highlighted to thedriver on the at least one display device.

The displayed images may be augmented by adjusting contrast levels ofthe captured images in response to a background color from the externalenvironment.

The system may further include an alert module to generate a speeddependent audible alert when immediate danger is detected to warn thedriver not to change lane when there is a risk of colliding with anobject.

If the video extraction logics detects variations between the frames, atracking sequence of an identified potential hazard may be activated insubsequent frames as a precautionary measure without issuing anindication to the driver while the vehicle is continuously driven in thesame lane.

The system may further include a spatial segmentation module and atemporal segmentation module to estimate spatial and temporal separationbased on relative traveling speed detected between two sequentialframes.

Spatial segmentation may be performed in each frame to improve accuracyof the video extraction logics or when large variations in thebackground between successive frames are detected.

Spatial and temporal segmentation and fusion may be adaptively performedby detecting the variation between successive frames to reducecomputational complexity and improve data processing speed.

The system may further include a variation computation module to executean adaptive frame skipping algorithm to decompose the successive framesto preclude error augmentation and propagating during the trackingsequence.

The video extraction logics may generate a hierarchy of temporalvariation representations with luminance consistency that describes theillumination variations in order to estimate and compensate for globalmotion.

If user interaction is imposed, spatial segmentation may be fused with amask before fusing with temporal segmentation.

BRIEF DESCRIPTION OF THE DRAWINGS

An example of the invention will now be described with reference to theaccompanying drawings, in which;

FIG. 1 is a screen representation of the display when a driver wishes tomake a right lane change;

FIG. 2 is a screen representation of the display when a driver wishesreverse park;

FIG. 3 is a screen representation of the display when adaptive contrastadjustments are applied comparing before the adjustment (left image) andafter the adjustment (right image);

FIG. 4 is a screen representation of the display that is adjustedoptimally based on ambient lighting conditions where, an approachingvehicle is highlighted while other vehicles in the background aresubdued;

FIG. 5 is a screen representation of the display illustrating thatvehicles too far away to cause concern are not highlighted

FIG. 6 is an Object tracking principle

FIG. 7 is a block diagram of video object extraction; and

FIG. 8 is a process flow diagram of the semantic video object (SVO)algorithm.

FIG. 9 is a block diagram of a detection system for a vehicle inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

Referring to the drawings, a detection system 5 for assisting a driverwhen driving a vehicle is provided. The system 5 generally comprises: aplurality of image capturing devices 98, an image processing module 95and multiple display devices 92. The plurality of image capturingdevices is mounted to the vehicle to capture images of the externalenvironment of the vehicle. The image processing module identifiespotential hazards in real-time from the captured images and superimposesinformation of the external environment relating to the identifiedpotential hazards. The image processing module 95 may be provided in acenter console 95 located between the seat 94 of the driver and seat 98of the front passenger. The multiple display devices are LCD panels 92which display images processed by the image processing module 95 inreal-time. The superimposed information is provided in an area of theLCD panels 92 to ensure visual clarity of the images of the externalenvironment.

The original exterior side mirrors on the vehicle are replaced by thesmall video cameras 98. By removing the original side mirrors, areduction in the drag coefficient of the vehicle is achieved whichimproves fuel efficiency There may be four to six video cameras 98mounted on a vehicle, for example, at the location of the B pillar 97also. The image from each camera 98 is captured at 25 frames per second(fps). Each image is therefore updated once every 40 ms. The visualinformation from the cameras 98 is input to a processor. The system 5extracts and identifies any potential hazards using video extractionlogics. Appropriate signs and signals are displayed to alert the driver.Various methods may be used to automatically defect the behavior ofdriver/Vehicle in anticipation of its intention thereby correctlypinpoint relevant objects of concern. This includes situations such aschanging lanes in different conditions and reverse parking. The videoextraction logics enable identification of nearby vehicles that maycause potential hazards. Visual and audible alerts are generated. Also,information: about approaching vehicle including traveling speed,relative speed to the driven vehicle, and estimated clearance time for asafe lane change are displayed to assist a safe lane change.

The LCD panels 92 are mounted on both sides inside the A-pillar 91 sothat the windscreen 90 is unobstructed. When the system 5 is operating,a first frame is captured and displayed on the LCD screen 92. The LCDscreen 92 is continuously updated until the ignition is switched off.The LCD panels 82 are adjustable with a little joystick for optimalviewing. There is also a memory setting which remembers the position ofthe driver's seat 94 and height of the steering wheel 43 if it is fittedwith electronic adjustment. The optimal LCD screen size, when viewedfrom the driving position, is about 6″. This is perceived by the driverto be slightly larger than the normal side view mirror.

Turning to FIG. 1, at cruising speed, the driver signals right andslight movement of the vehicle indicates to the system 5 that the driverintends to make a right lane change. Approaching vehicles 10 areextracted from the background using video processing technology andrelated information 11 is calculated, to alert the driver. A flashingred circle 12 highlights any incoming vehicle 10 to alert the driver.Information 11 about the vehicle 10 is displayed including: currentspeed is 64 km/h, relative speed is +4.7 km/h (the vehicle the driver isdriving is currently traveling at around 59 km/h) and based on currentspeed of each vehicle, the system 5 calculates a 1.6 second clearancefor a safe lane change

FIG. 2 depicts a reverse parking example. As the driver reverses topark, two potential hazards 20, 21 are defected by the system 5. A post20 is detected 0.3 m away from the rear end of the vehicle and a wall 21is detected 0.7 m away from the rear end of the vehicle. The closerobject is the post 20 which is highlighted by a larger arrow 22. Thesize of the arrow corresponds to the distance from the vehicle. Bothobjects 20, 21 detected in this example are stationary therefore noinformation about their relative movements is displayed. The informationwhich is displayed includes minimal distance (at the point closest tothe car). This information is in the form of both text and an indicatingarrow which changes size based on distance away from the vehicle. A reddash indicates that the vehicle has stopped hence no remaining time tophysically contact the objects is available for display. Otherwise, whenthe vehicle is moving, the time to reach each object is displayed.

To avoid causing unnecessary confusion to the driver, a maximum of twoobjects (of shortest distance) are visually identified at any giventime. This prevents the display from being overwhelmed with too muchinformation. The driver may set a predetermined distance to trigger anaudible alarm, for example, when the distance is less than thirtycentimeters.

The system 5 modifies the displayed image to the driver in several ways.These include: changing the colour of the displayed items (text,circles, etc) based on the background of the external environment,highlighting changes based on urgency and adaptive contrast adjustmentand glare reduction.

Turning to FIG. 3, under situations such as heavy fog, the contrastlevel of the display is automatically adjusted to improve viewingquality. For comparative purposes only, the let image 30 is an unalteredimage and the right image 31 has adjusted the contrast level of the leftimage to improve viewing quality. Also, glare from the headlamps ofvehicles behind the vehicle are reduced by selectively reducing thelight intensity on the display. Pixels detecting bright lights aremomentarily made less sensitive. In addition, the system 5 assists innight driving by optimally adjusting the display based on ambientlighting conditions.

Turning to FIG. 4, the vehicle 41 represented by the white triangle isfurther back than the vehicle 40 represented by the black triangle. Thevehicle 41 represented by the white triangle is subdued in thebackground so that the driver pays less attention to it whileconcentrating on the much nearer one (the vehicle 40 represented by theblack triangle). Referring to FIG. 5, other vehicles 50 which are toofar away from the vehicle to cause concern are not highlighted.

Typically, a driver would take a number of subtle actions prior tosignaling three seconds before making an anticipated lane change. Thesystem 5 senses the behavior of the driver in real-time in the followingways:

(1) yore frequent starring at the respective side mirror. For example,when the driver intends to make a right lane change, they wouldnaturally check the right mirror more often. This is detected by eyemovement sensing using sensor selection of an auto-focus camera.

(2) The system 5 defects more frequent minor movement of head to acertain side, for example, a slight turn towards the right beforesignaling to indicate the intention of the driver.

(3) Inexperienced drivers may have the tendency to steer the vehicleslightly when a change lane is desired. A comparison is made relative tothe contour of the driving lane. When an abnormal drift of the steeringwheel 93 is detected, this may also indicate the intention of the driverof to make a lane change.

(4) When the driver signals the left or right indicator, the system 5considers this to be a confirmation of the anticipated lane changeintention. Audible warnings can also be activated if danger from theadjacent lane is detected at the time.

There are several practical ways of sensing the driver behaviordescribed earlier. These Include facial detection by tracking eyemovement and head movement by a small low resolution camera fitted nearthe central rear vision mirror. This is an economical solution. Thesteering wheel may also be connected to a switch that closes when anabnormal drift is detected. This activates the warning unit that beginsto identify any potentially hazardous moving objects that are displayedby the side view mirror.

The system 5 is designed to proactively identify potential hazards whilecausing minimal distraction to the driver. Therefore, alerts are onlydisplayed when the driver intends to change lane. Any of the followingconditions trigger an alert by highlighting the object:

1) drift in steering causing the gap between the vehicle and the objectto decrease;

2) driver signals to indicate intention to turn towards the direction ofthe object;

3) motion detection showing more regular gazing at the mirror.

A speed dependent audible alert is generated when immediate danger isdetected so that the driver is warned not to change lane when there is arisk of colliding with another object. The traveling speed of theidentified object, along with its relative speed to the vehicle, iscalculated to give the driver adequate information to determine whethera lane change is safe. The actual spatial separation from the vehicle isonly used for calculation purposes but not displayed as the actualdistance is not of any practical use to the driver. From the relativespeed and spatial separation, an estimation on the time that takes forthe object to catch up with the vehicle can be calculated so that thedriver can be informed of how much time there is to cut in front of theobject. In this particular illustration, the object is assumed to travelat a variable speed and the information is constantly updated. Trackingof any identified object ceases once it disappears from threeconsecutive frames under the assumption that the object no longer posesany risk.

Referring to FIG. 6, each frame is compared with previous frames todetect relative movement of vehicles within range. Starting from thek.sup.th frame, the system 5 evaluates k-1 and k-2 to extract vehiclesthat have moved closer to the vehicle during the previous two frames.Four successive frames are illustrated in FIG. 6. k-1 observes an object(a nearby moving vehicle) which becomes closer than in frame k-2. Whenthe same object appears in frame k its position is compared to that ofk-1. The reduction in separation from the vehicle in the frame activatesa tracking sequence of the object in subsequent frames as aprecautionary measure and no indication to the driver is issued whilethe vehicle is continuously driven along the same lane. Based onrelative traveling speed detected between two sequential frames, anestimation of spatial and temporal separation can be calculated anddisplayed on the display. Spatial segmentation means partitioning theimage to a number of arbitrarily shaped regions, each of them typicallybeing assumed to constitute a meaningful part of the image, i.e. tocorrespond to one of the objects depicted in if or to a part of one suchobject.

Referring to FIGS. 7 and 8, a semantic video object (SVO) algorithm isdepicted. Accuracy in correctly identifying potentially hazardousobjects is vital since an inaccurate SVO containing any part of thebackground merged to the object or losing parts of the object canseverely affect the calculation of its relative position from thevehicle. Maintaining accuracy is a challenging issue for video sequencescontaining complex backgrounds in inner city or fast moving vehicles ona motorway. Accuracy can be improved by performing spatial segmentationin each frame. However, to reduce computational complexity and improvedata processing speed, the system 5 adaptively performs spatial 77, 78and temporal segmentation 82 and fusion 83 when necessary by defectingthe variations between successive frames. Spatial and temporalsegmentation is only performed when large variations in backgroundbetween successive frames are detected. Otherwise, the system 5 tracksthe previous boundary of the SVO.

The motion estimation and compensation module 71 of the system 5constantly searches for other potential hazards that may soon comewithin range. The spatial segmentation modules 72, 74 of the system 5also track the movement of all identified objects. For example, theidentified vehicle 10 may accelerate or decelerate and its movement istracked to determine its position in real-time relative to the vehicle.In such situations, the information is updated in real-time and anaudible sound alerts the driver when this vehicle is closing in at afaster rate.

The temporal segmentation module 82 is used to detect and track movingobjects such that any object moving towards the vehicle at a speedhigher than that of the vehicle's current speed can be identified withits movement tracked and compared with the previous frame; objectboundaries with pixel accuracy can be estimated in the process. At thesame time, unrelated foreground and background areas can be extractedand eliminated while tracking relevant moving objects. Next, objectdetection with spatio-temporal fusion motion trajectories 83 from themoving object currently being tracked will generate a hierarchicaldescription of the object's movement from trajectory data for the SVO.

The SVO identifies the initial object contour of any object that appearsin the frame, any moving object is extracted from the (stationary)background and tracked on a frame-by-frame basis. An adaptive frameskipping algorithm is executed in the variation computation module 70 todecompose video sequence/successive frames to preclude erroraugmentation and propagating during the object tracking process.

Turning to FIG. 7, Case 1 shows the normal cruising condition whennearby moving objects are tracked by the spatial segmentation module 72.

If the vehicle is pulled over, Case 2 indicates the situation where thevehicle remains stationary and the vehicle is expected to be driven awayat any time so that the system continues monitoring the surrounding incase the vehicle is driven back onto the road. In this case, only oneside of the system is activated.

Case 3 allows manual intervention by the driver, particularly useful insituations such as overtaking when the vehicle suddenly accelerates.Depending on the vehicle driven, it may already have a user interfacethat enables swift manual intervention of the system, such as bysuspending all audible warning signals through touching a large area ofa touch screen that corresponds to temporary suspension which requiresSVO selection by the user 73. The main purpose of this feature is toallow the driver quick access to manually disabling warnings that areunnecessary and may cause unnecessary attention during certain drivingconditions. In this case, the automatic hazard detection system caneither be temporarily disabled or any nearby vehicle such as the one tobe overtaken can be selected to be tracked so that information onrelative speed and distance can be calculated in real-time.

Case 4 shows any immediate dangers detected and highlighted, informationfrom the previous frame is used for continued tracking of nearby vehiclecausing elevated hazard. At the same time, all other vehicles within theframe are still tracked in case they too cause danger at a later time.

Case 5 is activated during parking. Boundary tracking 78 is used formonitoring slow-moving objects or when the vehicle is reversing at lowspeed. Morphological differences are examined by compared with theprevious frame for tracking of slow movement with high precision. Thisis accomplished by multiplexing the outputs from spatial-temporal andboundary tracking to generate the SVO parameter for the current frame asseen on the display using the multiplexers 80, 81.

The delay 79 tracks the frame sequence in such a way that the currentframe becomes one previous in the context of a continuously updatingvideo sequence.

The system 5 first estimates and compensates for global motion based ona simplified linear model. The simplified linear model is a model forvideo segmentation. To estimate the parameters for the simplified linearmodel, a hierarchical block matching and least square approximation areused. The SVO algorithm generates a hierarchy of temporal variationrepresentations with luminance consistency that describes theillumination variations. The variation between successive frames N andN+1 is derived using the sum of the second derivative of the frameluminance of each pixel. The SVO algorithm extracts information betweenthese successive images of any given sequence on temporal variationsusing motion and variations of illumination parameters.

The system 5 performs SVO extraction according to two criteria. Firstly,if the variation between successive frames k-1 and k is larger than apredefined threshold Tv, spatial and temporal segmentation is performedand fused. This obtains an accurate SVO. Otherwise, the system 5 tracksthe boundary of the extracted SVO in frame M to produce an SVO for frameN−1. This reduces the computational complexity and amount of processingtime.

Secondly, if user interaction is imposed, spatial segmentation must befused with the is mask 75 of the SVO that the driver provides or thesystem 5 obtains from frame k-1 before fusing with temporalsegmentation. Temporal segmentation defects moving regions from thebackground of each frame. The system 5 then removes the connectedcomponents with an area less than a predefined threshold (a constantpercentage of the frame size) by using a morphological openingoperation. The predefined threshold is not fixed for classifying thedetected motion change as a moving vehicle. Generally, the purpose ofidentifying the foreground from the background is to only track objectsof potential concern. For example, a nearby vehicle that approaches fromthe adjacent lane should take up at least one block within a givenframe. The representation of such a vehicle should be different fromthat of the next frame. Such difference is identified and used to trackthe movement of the object. Holes inside the moving objects areas arethen removed by sequentially performing opening and closing operations.Finally, boundary tracking is performed when the variation betweensuccessive frames is below the predefined threshold.

In FIG. 8, frame N is captured during any given time while driving undernormal conditions. It updates to the next frame N+1, which may or maynot contain any useful information compared to frame N. When no hazardis detected, as expected for the majority of time when the vehicle iscruising, the process continually repeats itself 92, 93. Motionestimation continues to defect any change in environment. The detectionof the driver's response 95, for example, a lane change indicated, willsearch and detect any object of potential hazardous nature 100.

An image encoder (for example, an encoder that performs DOT orKarhunen-Loeve transform) divides each frame into blocks. Each blockcontains vectors portraying the manifestation spaces of the trackedobjects. As the vehicle moves, the background changes while the keyfeatures should not differ significantly between any two successiveframes. The sequence of frames therefore have a high temporalredundancy. Consequently, a continuously strong inter-frame referenceexists in the sequence. The correlation of inter-frame reference in eachframe and its temporal changes are used to compute the probability of asudden and significant change within a block of several consecutiveframes, i.e. in a fraction of a second's time where a substantial changeis anticipated such as when a fast moving vehicle is closing in.Temporal segmentation relies on motion information acquired fromconsecutive frames that does not have any information about the spatialdomain. When the background changes rapidly, such as negotiating a sharpbend, the background does not appear to be static when compared to theprevious frame, with almost identical color patterns but very differentshapes, are updated from frame to frame by embedding the residualdifference between the background in the previous and current frames.Temporal segmentation splits the captured video info scenes (thebackground) and shots (successive frames with variation of objects'relative positions to the vehicle). Spatial segmentation splits eachframe into blocks. Spatial segmentation is performed according toinformation of the frame such as motion features, color and texture. Thepurpose of performing spatial-temporal segmentation is to separate theforeground and background from each other. Therefore spatial-temporalsegmentation enables extraction of any relevant objects from the ambientbackground in the external environment of the vehicle.

If will be appreciated by persons skilled in the art that numerousvariations and/or modifications may be made to the invention as shown inthe specific embodiments without departing from the scope or spirit ofthe invention as broadly described. The present embodiments are,therefore, to be considered in all respects illustrative and notrestrictive.

The video image processing system with a SoC (System-on-Chip)implementation, when used in conjunction with CCD cameras and LCDpanels, is expected to operate very reliably over long periods of timewith virtually no servicing required since it does not have any movingmechanical parts. However, in the unlikely event that the system mayfail, the system can be designed to automatically activate a pair ofmechanical side mirrors that are normally concealed as an integral partof the A-pillar. As these side mirrors are only deployed in the event ofan emergency, similar to the case of temporarily replacing a flat tyrewith a spare stored in the trunk (which is not intended for normaldriving). The backup mirrors can be made small enough to be fitted intothe A-pillars without impairing their physical strength or visualappearance of the A-pillars.

1. A detection system for assisting a driver when driving a vehicle, thesystem comprising: a plurality of image capturing devices mounted to thevehicle to capture images of the external environment of the vehicle; animage processing module to identify potential hazards in real-time fromthe captured images and to superimpose information of the externalenvironment relating to the identified potential hazards; and at leastone display device to display images processed by the image processingmodule in real-time; wherein the superimposed information is provided inan area of the display device to ensure visual clarity of the images ofthe external environments wherein the image processing module comprisesvideo extraction logics to extract and identify potential hazards, thevideo extraction logics detecting the variations between successiveframes captured by the plurality of image capturing device; and whereinif the video extraction logics detects variations between the frames, atracking sequence of an identified potential hazard is activated insubsequent frames as a precautionary measure without issuing anindication to the driver while the vehicle is continuously driven in thesame lane.
 2. The system according to claim 1, wherein the imageprocessing module comprises video extraction logics to extract andidentify potential hazards, the video extraction logics detecting thevariations between successive frames captured by the plurality of imagecapturing device.
 3. The system according to claim 1, further comprisinga camera for tracking eye movement and head movement in order to detectthe behaviour of the driver in anticipation of intention, and thedetected behaviour is used by the image processing module to determineobjects of concern that are highlighted to the driver on the at leastone display device.
 4. The system according to claim 1, furthercomprising a switch operatively connected to a steering wheel of thevehicle to detect the abnormal drift of the vehicle, and the detectedabnormal drift is used by the image processing module to determineobjects of concern that are highlighted to the driver on the at leastone display device.
 5. The system according to claim 1, wherein thedisplayed images are augmented by adjusting contrast levels of thecaptured images in response to a background color from the externalenvironment.
 6. The system according to claim 1, further comprising analert module to generate a speed dependent audible alert when immediatedanger is detected to warn the driver not to change lane when there is arisk of colliding with an object. 7.-11. (canceled)
 12. The systemaccording to claim 2, wherein the video extraction logics generates ahierarchy of temporal variation representations with luminanceconsistency that describes the illumination variations in order toestimate and compensate for global motion.
 13. The system according toclaim 2, wherein if user interaction is imposed, spatial segmentation isfused with a mask before fusing with temporal segmentation.